Automated Analysis of Computer-Generated Software Usage Protocols: An Exploratory Study

Ilighly-intcractive computer software can potentially help users think and work more effectively. To realize this potential, sotiware developers should understand the cognitive processes involved in the tasks being performed and the ways users interact with the software to accomplish the tasks. Gathering data about software usage-<:alled protocols-is costly in several ways, including preparing representative test scenarios, fmding suitable subjects, training personnel to administer the tests and record the protocols. and collecting and coding the protocols. Similarly, analyzing protocols can be tedious, often done manually by skilled researchers. Because of their high costs, protocol studies frequently consist of a few subjects tested while performing synthetic tasks in an artificial setting. The value of these studies is limited both for software developers and researchers in human-computer interaction. This paper describes a method used to collect and analyze the protocols of a large number of subjects performing tasks in a naturalistic setting. An interactive computer program was developed as a testbed for this study. It contained an automatic tracker that unobtrusively collected protocol records of users' interactions with the program. L sers' strategies in working with this program were modeled as a formal grammar, and a parser was devised, based on the grammar, to analyze protocol records produced by the program. C sers' behaviors and strategies of working with the program were examined and characterized, based upon the parsed protocol data. A graphical structure editor was created as .the testbed for this study; it assists in expository writing, such as technical journal articles, with special emphasis on the exploratory and organizational phases of the writing process. This paper discusses lessons learned in devising the grammar to model users' writing sessions with the editor, in building and refuring the parser, and in analyzing the protocol records for 112 sessions collected from 29 subjects.